fix(tests): make 3 CI-only-failing tests hermetic (closes #74)#75
Merged
Conversation
… clean CI The two "akm CLI availability" fallback tests passed on dev machines but failed on clean CI because they depended on machine state: 1. Bundled-CLI preemption: akm 0.8.0 added bundled akm-cli resolution (getResolvedAkmDetails/getBundledAkmCommand) tried before PATH/fallbacks. The real bundled opencode/node_modules/akm-cli/dist/cli.js exists on CI and was probed/selected, shadowing the candidate each test asserts on. Both tests now set AKM_OPENCODE_IGNORE_BUNDLED_CLI=1 (the existing default-off opt-out) so the bundled candidate is skipped. 2. Hidden filesystem dependency: probeCommand() does a real existsSync() on absolute candidate paths even when execFileSync is mocked. The fallback test never created ~/.local/bin/akm and never sandboxed HOME, so on clean CI the candidate was skipped. It now sandboxes HOME to a mkdtemp dir and creates $HOME/.local/bin/akm with a shebang so existsSync passes deterministically. 3. os.homedir() snapshot: getPathAkmCandidates() builds the user-local fallback from os.homedir(), which snapshots HOME at startup and ignores a later process.env.HOME change. A node:os module mock now delegates to the real os but makes homedir() honor the current process.env.HOME, so the HOME sandbox actually redirects the probed path into the temp dir (never the real ~/.local/bin/akm). All writes stay inside the mkdtemp sandbox. The fallback test's mock also returns an out-of-range semver for any non-target --version probe, defeating both the execFileSync shim's "synthesize 0.8.9" behavior and a stale resolvedAkmCommand left cached by a prior test. Assertions unchanged; full file 158/158 passes, including under a simulated clean-CI env (empty HOME, no akm on PATH, bundled cli.js present). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "session-start injects hints and curated context" test wrote its fake
`akm config set` helper to be run by a bare `node` resolved from the hook's
PATH (`${binDir}:/usr/bin:/bin`). That only works on hosts that ship
`/usr/bin/node` (most dev boxes). On clean CI the suite runs under Bun with
no Node on PATH, so the helper exited non-zero, `akm config set` failed, the
config file stayed `{}`, and the `config.defaults.agent` /
`config.profiles.agent.claude` assertions failed.
Pin the fake helper to the test suite's own interpreter (process.execPath,
i.e. the Bun running `bun test`) instead of a bare `node`. Bun runs `.cjs`
CommonJS modules natively, so the config write now succeeds regardless of
whether Node is installed.
Note: the Claude SessionStart hook (claude/hooks/akm-hook.ts) resolves akm
via AKM_LOCAL_BUILD_CLI then PATH `akm` only — it has NO bundled-akm-cli
node_modules resolution, so the bundled-CLI hypothesis did not apply here.
The real environment dependency was Node availability for the test's fake.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…he between tests The 'prefers ~/.config/opencode/.bin/akm' test still failed on clean CI: - getPathAkmCandidates used bare os.homedir() (snapshots HOME at process start, ignores later changes) for the ~/.local/bin fallback, so it probed the real CI home instead of the test's sandbox HOME. Now uses process.env.HOME || os.homedir(), matching getConfigNodeModulesAkmCommand (standard POSIX, sandboxable). - resolveAkmCommand caches the resolved command at module level and short-circuits on a still-valid cached command, making resolution tests order-dependent. Add __resetResolvedAkmForTests() and call it in the suite beforeEach so each test resolves fresh under its own sandboxed env. Local gate green: integration 324/0, evals exit 0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
getConfigNodeModulesAkmCommand reads XDG_CONFIG_HOME || $HOME/.config, but the test wrote its config bin under $HOME/.config directly. On CI XDG_CONFIG_HOME is set elsewhere, so the resolver probed the wrong dir and never found the config candidate (passed locally only because XDG_CONFIG_HOME was unset). Pin XDG_CONFIG_HOME to the sandbox and write the bin there. Verified robust by running with a bogus preset XDG_CONFIG_HOME (reproduces the CI condition). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Eval diff
Metric deltas
Plugin surfaceNo surface changes. No regressions. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes the 3 tests that passed locally but failed on clean CI hermetic. Closes #74.
opencode-plugin.test.ts (2 tests) — commit e3ecc99
AKM_OPENCODE_IGNORE_BUNDLED_CLI=1so the real bundled akm-cli can't preempt the candidate under test.HOMEto a temp dir + create the probed fallback file there (realexistsSyncinprobeCommandmade these env-dependent).node:oshomedir()to honor the currentprocess.env.HOME— Bun/Node snapshot HOME at startup, so a HOME env change alone didn't moveos.homedir()(whichgetPathAkmCandidatesuses). This is why they passed on dev boxes (real~/.local/bin/akm) and failed on clean CI.claude-plugin.test.ts (1 test) — commit fadad21
AKM_LOCAL_BUILD_CLIor PATH only). The test's fakeakmshelled out to a barenodeforconfig set; CI runs under Bun with no Node on PATH, so the config write failed and assertions saw{}.process.execPath(the Bun running the suite), which is guaranteed present. Bun runs.cjsnatively.Local gate green: integration 324/0, evals exit 0. Data safety: all writes under mkdtemp temp dirs, cleaned up; nothing touches real
$HOME.🤖 Generated with Claude Code